VMware→KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes#13656
Conversation
Operator-driven warm VMware-to-KVM migration built on VDDK and VMware Changed Block Tracking (CBT). Keeps a source-equivalent KVM-side replica current through delta cycles, then finalizes with virt-v2v in-place at cutover. Supports both filesystem (qcow2) and native Ceph/RBD (raw image) primary storage targets, with preflight validation, async lifecycle operations, persisted migration state, and Import/Export UI integration. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the existing VDDK + virt-v2v VMware-to-KVM import path so the converted disk can be written straight into an RBD pool as a raw image, with in-place virt-v2v finalization. Adds host-capability detection for qemu-img RBD support, RBD qemu copy, virt-v2v in-place, and direct RBD import, advertised via ReadyCommand and reconciled by the agent manager. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Extends the KVM unmanaged import / manage-volume flow so an existing raw RBD image can be adopted directly as a VM root volume, including the RBD-format check in the volume orchestrator and the libvirt volume inspection path. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Unifies the host-capability keys that the CBT warm-migration and VDDK direct-RBD-import features introduced independently under different names, using feature-neutral names that describe what is probed: host.vmware.cbt.support -> host.vddk.blockcopy.support host.vmware.cbt.in.place.finalization.support -> host.vddk.blockcopy.inplace.finalization.support host.vmware.cbt.rbd.support -> host.vddk.blockcopy.rbd.support host.qemu.img.rbd.support -> host.qemu.rbd.support host.virt.v2v.inplace.support / host.virtv2v.in.place.version -> host.virtv2v.inplace.* Drops the redundant host.rbd.qemu.copy.support alias (identical to host.qemu.rbd.support) and its hostSupportsRbdQemuCopy() delegate. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13656 +/- ##
============================================
+ Coverage 19.64% 19.76% +0.11%
- Complexity 19789 20192 +403
============================================
Files 6368 6420 +52
Lines 575109 582201 +7092
Branches 70371 71221 +850
============================================
+ Hits 112994 115060 +2066
- Misses 449829 454652 +4823
- Partials 12286 12489 +203
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@blueorangutan package kvm |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18626 |
…isk import The "Manage Instances" source dropdown offered "Import QCOW2 image from Local/Shared Storage", but importVm now adopts an existing ROOT disk from Filesystem, NetworkFilesystem, SharedMountPoint and RBD (raw) pools, so the QCOW2-only wording is misleading. Rename both entries and their wizard title/description to "Import Instance using existing ROOT disk from Local/Shared Storage", across the en, pt_BR and te locales. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
docs/vmware-cbt/architecture.md and docs/vmware-cbt/README.md were added without the Apache license header, which failed the RAT license check and the pre-commit insert-license hook. Add the standard ASF header (identical to the one already present in docs/vmware-cbt-migration.md). Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
…ey names State Apache CloudStack 4.24.0.0 as the target release (schema delivered via the 4.24.0.0 upgrade path, with a note that the DDL currently rides in schema-42210to42300.sql pending the main version bump), and replace the pre-consolidation host-capability key names with the current host.vddk.blockcopy.* / host.virtv2v.inplace.version names. Signed-off-by: Andrija Panic <andrija.panic@gmail.com>
Staged VMware-to-KVM imports (virt-v2v conversion to an NFS temporary location followed by a copy to the destination pool) can now target Linstor primary storage: - Linstor added to the staged-conversion destination pool types, so destination pool discovery considers cluster- and zone-wide Linstor pools alongside NFS and RBD. - The import host is validated to have access to each Linstor destination pool (it must be a LINSTOR satellite connected to the pool); the RBD qemu-copy capability check is preserved and the validation renamed to validateStagedImportHostSupport. - The KVM-side converted-disk move already dispatches by pool type to LinstorStorageAdaptor.copyPhysicalDisk, which spawns the resource from the pool's resource group and qemu-img converts the qcow2 into the DRBD block device; converted-disk metadata for Linstor pools now reports the pool UUID and volume name instead of parsing NFS mounts (Linstor pools have no local mount path). The imported volume keeps the copy-time UUID as its path; the Linstor driver addresses existing volumes by path, so later volume lifecycle operations (attach, resize, snapshot, delete) work unchanged.
A standalone design/validation document covering the Linstor-destination portion of this series, written so LINSTOR/DRBD maintainers can review the storage-side behavior: the RAW_BLOCK_DEVICE target model, pre-created resources with qemu-img convert -n / nbdcopy, in-place finalization on the local DRBD device, resource-naming constraints, delta-copy semantics, host prerequisites, the support matrix, and the lab/end-to-end validation performed.
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18633 |
| import static com.cloud.host.Host.HOST_VIRTV2V_INPLACE_SUPPORT; | ||
| import static com.cloud.host.Host.HOST_VIRTV2V_INPLACE_VERSION; | ||
| import static com.cloud.host.Host.HOST_VIRTV2V_VERSION; | ||
| import static com.cloud.host.Host.HOST_VDDK_BLOCKCOPY_INPLACE_FINALIZATION_SUPPORT; |
There was a problem hiding this comment.
Nit: these new static imports (HOST_VDDK_BLOCKCOPY_*) break the existing alphabetical import order (they land after HOST_VDDK_VERSION instead of before HOST_VDDK_LIB_DIR).
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| import org.apache.commons.collections.CollectionUtils; |
There was a problem hiding this comment.
Nit: this imports the legacy org.apache.commons.collections.CollectionUtils (also in LibvirtVmwareCbtCleanupCommandWrapper.java, LibvirtVmwareCbtCutoverCommandWrapper.java, LibvirtVmwareCbtPrepareCommandWrapper.java), while other new files in this PR use commons.collections4. Worth picking one.
| "message.desc.import.ext.kvm.wizard": "Import libvirt domain from External KVM Host not managed by CloudStack", | ||
| "message.desc.import.local.kvm.wizard": "Import QCOW2 image from Local Storage of selected KVM Host", | ||
| "message.desc.import.shared.kvm.wizard": "Import QCOW2 image from selected Primary Storage Pool", | ||
| "message.desc.import.local.kvm.wizard": "Import an Instance using an existing ROOT disk from the Local Storage of the selected KVM host", |
There was a problem hiding this comment.
Nit: this changes "selected KVM Host" to "selected KVM host" (lowercase) and the sibling string below drops "Storage Pool" to "Storage pool", a casing regression from the previous strings.
There was a problem hiding this comment.
Labeling is SO incosistent (plsm check that file 5000 times "host" and another 5000 times Host - I'm lazy - but feel free to change it if you want
I'm more interested in actual testing the value of the PR than semantically
|
@rp- — this PR now also adds Linstor as a destination for the To make that easy there's a standalone write-up committed in the PR: The Linstor-side design points I'd most like sanity-checked:
Validated on a 3-node LINSTOR/DRBD 9 lab: byte-identical disk copies |
|
Great to see this! I can't test it (don't have any env or even VMware), but a review will be though. It's a big PR with a lot of lines to check. I assume you tested this? Can it break any existing things? |
Tested "offline" but also being tested "more" properly as well. Smoke tests etc - comes later - should not break anything |
EL9-family distributions (Rocky/RHEL/Alma 9) install virt-v2v-in-place in /usr/libexec, outside $PATH. Probing the bare binary name therefore reported host.virtv2v.inplace.support=false, which cascaded to host.vddk.blockcopy.inplace.finalization.support and host.vddk.rbd.direct.import.support and silently disabled the direct-VDDK and warm-CBT block-device import paths on the most common production platform. Resolve the executable via LibvirtComputingResource.getVirtV2vInPlaceBinary() ($PATH first, then /usr/libexec) and use the resolved binary in the capability probe, the version detection, and every finalization invocation (direct convert and CBT cutover).
Direct VDDK import into an RBD pool copied each disk, then finalized with virt-v2v-in-place fed a domain XML whose disks were native <disk type='network' protocol='rbd'> sources with a cephx <auth> element referencing a libvirt secret. virt-v2v's '-i libvirtxml' input runs without a libvirt connection, so the secret could not be resolved and libguestfs attached no drives, failing with 'you must call guestfs_add_drive before guestfs_launch' (observed on EL9). Serve each RBD image over a temporary localhost qemu-nbd bridge and point the finalization XML at the NBD endpoints instead - the same approach the CBT cutover already uses - so no secret needs resolving. The bridges are torn down on exit.
Copying a converted disk into a freshly created thin block-device target (Linstor/DRBD) fully allocated the volume: qemu-img convert and nbdcopy do not assume a block-device destination is zeroed, so they wrote out every zero block. A 30 GiB volume holding ~2 GiB of data became 99.99% allocated on all replicas, exhausting the thin pool. Pass -n --target-is-zero to qemu-img convert and --destination-is-zero to nbdcopy for pre-created block-device targets. Also fix the QemuImg --target-is-zero capability detection: newer qemu (10.x on EL9) no longer lists the flag in 'qemu-img --help' while still supporting it, so probe the convert option directly - this is also why the Linstor adaptor's existing zero-skip (copyPhysicalDisk) silently stopped engaging.
…deletes When a CBT migration is cancelled, its initial/delta sync data-plane copy is not aborted by the cancel itself, so it can still be writing into - and holding open - the very target volumes the cleanup then tries to delete. The surviving copy also keeps burning VDDK/network/storage on a dead migration. Before deleting the marker-guarded target images/volumes, kill the in-flight copy processes whose command line references this migration's marker (matches the nbdcopy / qemu-img convert / qemu-io processes; the nbdkit --run parent exits with its child), then delete with a short retry to ride out the brief window where the device is still held open (DRBD demotion, qemu teardown).
…BT states - importVm (VMware to KVM): validate the data-disk offering mappings right after the source instance is discovered, before any conversion starts. The same count check previously ran only at import time, after every byte of the source disks had already been copied and converted (~20 wasted minutes per attempt in lab testing), and a validation failure at that late stage also orphaned the already-copied direct-import volumes. - startVmwareCbtMigration: same pre-flight against the discovered source disks, so a missing datadiskofferinglist fails before the initial full sync instead of at cutover import. - markMigrationFailed: do not overwrite a terminal migration state - a cancel racing with an in-flight sync previously flipped Cancelled to Failed when the aborted sync's failure landed afterwards.
…py for RBD) Two data-plane efficiency improvements for the agent's own nbdkit full-disk copies (cold direct import and CBT initial full sync). Neither touches the staged virt-v2v conversion path, which drives its own VDDK input. - VDDK NBD compression: the nbdkit vddk plugin now runs with compression=<value> (new agent.properties key vddk.nbd.compression, default fastlz; none/zlib/fastlz/skipz). In extent-less setups (e.g. NFS-backed datastores) VDDK reports the whole disk as allocated and the full nominal size crosses the network; fastlz compresses both zero regions and real data on the ESXi side at low CPU cost. Measured on a 30 GiB disk: 32.4 GB over the wire uncompressed vs 2.65 GB with fastlz. - nbdcopy over qemu-nbd for RBD full copies: cold-direct-to-RBD and the CBT initial full sync to RBD previously used a single-stream qemu-img convert (RBD images are created inline by qemu-img, so they were never pre-created). They now pre-create the raw RBD image (qemu-img create), serve it over a localhost qemu-nbd bridge, and copy NBD-to-NBD with multi-connection nbdcopy - the same bridge approach the CBT cutover uses - falling back to qemu-img convert when nbdcopy is unavailable. The fresh image reads as zeros, so --destination-is-zero keeps it sparse. The Linstor block-device path already used nbdcopy directly; delta cycles are unchanged (a range copier is required there).
rp-
left a comment
There was a problem hiding this comment.
The direct Linstor implementation of listPhysicalDisks looks fine.
BUT the new code always has the assumption that Linstor volumes are zeroed,
which is sadly not true, even more so as I'm working on getting shared disk support ready and that will only work on LVM-Thick (which doesn't guarantee zeroed devices).
So I think we need some checks for target-is-zero and if the block devices supports it,
I guess this could also be the case for other block storage providers.
The direct-VDDK and CBT initial-sync copies into a pre-created block-device target passed --target-is-zero / --destination-is-zero unconditionally, assuming a freshly created volume reads back as zeros. That holds for thin providers (LVM-thin, ZFS, diskless) but NOT for thick ones (e.g. LVM-thick), where a new volume can expose stale data from a previously deleted volume - skipping the zero-writes would leak that stale data into the unwritten regions of the migrated disk. Add KVMStoragePool.isVolumeZeroInitialized(volumeName) (default false) and implement it for Linstor via the existing LinstorUtil.resourceSupportZeroBlocks provider check (true only for LVM_THIN / ZFS / ZFS_THIN / DISKLESS). The copy paths now skip zeros only when it returns true; otherwise they write the zeros (no-create into the device, but fully). This mirrors the check the staged copyPhysicalDisk path already uses, and the default-false makes any future block backend opt in explicitly. RBD is unaffected (qemu-img creates the image itself, always zeroed). Raised in review by the LINSTOR plugin maintainer (shared-disk support lands on LVM-thick).
|
Good catch @rp- — fixed in da57d6a. Added KVMStoragePool.isVolumeZeroInitialized() (default false) so zero-skipping only happens when the backend guarantees it; Linstor implements it via the existing resourceSupportZeroBlocks provider check, so LVM-thick correctly writes zeros. Made it a generic pool capability as you suggested, so other block providers can opt in. Thanks for flagging the shared-disk / LVM-thick case. |
|
@blueorangutan package kvm |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18650 |
…versions When importing a multi-disk VMware VM directly onto an RBD or Linstor pool, the converted disks are discovered from the finalized libvirt domain, whose device order (vda, vdb, ...) need not match the source disk order. sanitizeConvertedInstance copied the source disk IDs onto the converted disks by list index, which swapped the root and data disks: the VM's root volume ended up on a data disk and the guest could not boot its OS. The identical source imported to NFS was unaffected. Pair each converted disk to its source disk by the creation position embedded in the pool image name (<uuid>-disk-NNN for RBD, <uuid>-dNN for Linstor) instead of by list index, falling back to index pairing for OVF conversions whose images carry no such position. Adds unit tests for reversed domain order, OVF fallback, and in-order cases.
Extends the KVM unmanaged-import / manage-volume adoption path so an existing raw Linstor (DRBD) volume can be adopted as a VM ROOT or DATA volume, mirroring the existing RBD adoption: - LibvirtCheckVolumeCommandWrapper: Linstor added to the supported pool types and routed through the raw-block-device inspection path (qemu-img on the /dev/drbd device) instead of checkQcow2File, which rejects a raw device. - LibvirtGetVolumesOnStorageCommandWrapper: Linstor added to the qemu-img supported types (data-volume listing/adoption); it inspects the local device path directly, no rbd: URI. - VolumeImportUnmanageService.SUPPORTED_STORAGE_POOL_TYPES_FOR_KVM: add Linstor so standalone data-volume import/unmanage accepts it. - VolumeOrchestrator.getSupportedImageFormatForCluster: RBD or Linstor => RAW. Unit tests mirror the RBD cases (CheckVolume via /dev/drbd device path, VolumeOrchestrator Linstor->RAW).
The generic IS_LOCKED signal comes from a host-local qemu-img file lock, which for Linstor/DRBD does not reflect a resource attached to a running VM on another node (unlike RBD's cluster-wide exclusive lock). Add a KVMStoragePool.getVolumeInUseNode() capability (default null), implemented for Linstor via the existing LinstorUtil.isResourceInUse cluster-wide InUse query, and OR it into the IS_LOCKED detail in both the CheckVolume (root) and GetVolumesOnStorage (data) adoption paths, so a volume still in use anywhere in the cluster is refused with a clear reason.
|
@blueorangutan package kvm |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with kvm SystemVM template(s). I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 18653 |
Description
This PR makes Ceph/RBD and Linstor first-class targets for KVM VM onboarding and adds an operator-driven warm VMware→KVM migration path. It bundles the following (Ceph/RBD plus Linstor as block-storage destinations):
1. VMware CBT → native RBD/qcow2 warm migration to KVM
An operator-driven, warm VMware-to-KVM migration built on VDDK and VMware Changed Block Tracking (CBT). It maintains a source-equivalent KVM-side replica kept current through repeated delta/sync cycles, then finalizes the guest with virt-v2v in-place at cutover — bounding downtime to the cutover window instead of a full cold copy.
qcow2) and native Ceph/RBD (raw image) primary storage.startVmwareCbtMigration,syncVmwareCbtMigration,cutoverVmwareCbtMigration,cancelVmwareCbtMigration,deleteVmwareCbtMigration,listVmwareCbtMigrations,checkVmwareCbtMigrationPrerequisites.VmwareCbtMigrationManagerImpl(+VmwareCbtMigrationServiceImpl, cutover policy), backed by new VOs/DAOs and a DB schema migration.docs/vmware-cbt/(architecture + README) anddocs/vmware-cbt-migration.md.2. Direct VDDK VMware import into Ceph/RBD — the cold (one-shot) path
The cold counterpart to the warm CBT migration in (1): extends the existing
VDDK + virt-v2v VMware→KVM import — which previously landed only on NFS/
qcow2(direct, or via a staged file) — so the converted disk is written straight into an
RBD pool as a raw image, with in-place virt-v2v finalization and no intermediate
qcow2-on-NFS staging copy. (The warm path in (1) reuses this same full-disk copy for its
initial sync.) Adds host-capability detection for qemu-img RBD support, RBD qemu-copy,
virt-v2v in-place, and direct-RBD import — advertised via
ReadyCommandand reconciledby the agent manager.
3.
importVmadoption of existing RBD (Ceph) root volumes and Linstor root + data volumesExtends the KVM unmanaged-import / manage-volume flow so an existing raw RBD image can be adopted directly as a VM ROOT volume. Adds RBD-aware format handling in
VolumeOrchestrator(RBD ⇒RAWinstead of the hardcodedQCOW2) and the RBD branch in the libvirt volume-inspection path (LibvirtCheckVolumeCommandWrapper), soCheckVolumeCommandsucceeds on RBD instead of failing with "Disk not found or is invalid".Also extends the same adoption path to Linstor (DRBD) primary storage, for both ROOT and DATA volumes: an externally prepared Linstor volume (e.g. landed by a migration/conversion tool) can be adopted without any data copy. Linstor volumes are raw local block devices, so volume inspection runs qemu-img against the
/dev/drbd/…device path (LibvirtCheckVolumeCommandWrapperfor ROOT,LibvirtGetVolumesOnStorageCommandWrapper+ the volume import/unmanage gate for DATA), andVolumeOrchestratorrecordsRAW. Because a host-local qemu-img file lock cannot see a DRBD resource that is Primary/open on another node (unlike RBD's cluster-wide exclusive lock), adoption additionally consults Linstor's cluster-wideInUsestate — newKVMStoragePool.getVolumeInUseNode()capability (default no-op), answered via the existingLinstorUtil.isResourceInUse— and refuses a volume attached to a running VM anywhere in the cluster. Volumes are adopted under their existing resource names (Linstor cannot rename; 48-char limit).4. Consolidate KVM host-capability probe names across the RBD features
Because features (1) and (2) introduced host-capability keys independently, this commit unifies them under feature-neutral names that describe what is probed, and drops a redundant alias (
host.rbd.qemu.copy.support, identical tohost.qemu.rbd.support) and its delegate.5. Linstor (DRBD) primary storage as a migration destination
Extends all three VMware→KVM paths above — staged import, direct VDDK import, and warm CBT migration — so the converted disks can also land on Linstor primary storage, not just Ceph/RBD. A generic
RAW_BLOCK_DEVICEtarget type covers "write RAW into a host-local block device provided by the storage adaptor", so the same code paths serve Linstor (and lay groundwork for other block backends).qemu-imgcannot create a DRBD device), copies withqemu-img convert -n— ornbdcopywhen the host has it — into the local/dev/drbd/…device, and finalizes withvirt-v2v-in-placefed a<disk type='block'>domain XML (no qemu-nbd bridge needed for a local device).qemu-img convert -n -S 0), avoiding a temp-file round-trip.-Oflag from thevirt-v2v-in-placeinvocation (fixes in-place finalization on virt-v2v < 2.5, e.g. Ubuntu 24.04) — which benefits the RBD path too.6. Data-plane efficiency and correctness for the agent-driven copies
These apply to the agent's own nbdkit/VDDK full-disk copies (cold direct import and CBT initial full sync); they do not touch the staged virt-v2v conversion path.
compression=<value>(newvddk.nbd.compressionagent property, defaultfastlz;none/zlib/fastlz/skipz). On datastores that don't expose block allocation to VDDK, the full nominal disk size otherwise crosses the network;fastlzcompresses both zero regions and real data on the ESXi side at low CPU cost (measured on a 30 GiB disk: 32.4 GB → 2.65 GB over the wire).nbdcopyfor RBD full copies — cold-direct-to-RBD and the CBT initial full sync to RBD now copy withnbdcopyover a localhostqemu-nbdbridge (pre-create the raw image, serve it locally, copy NBD-to-NBD) instead of a single-streamqemu-img convert, falling back toqemu-img convertwhennbdcopyis unavailable. The Linstor block-device path already usednbdcopydirectly; CBT delta cycles are unchanged (a range copier is required there).--target-is-zero/--destination-is-zero) only when the backend guarantees a freshly created volume reads back as zeros, via a newKVMStoragePool.isVolumeZeroInitialized()capability (Linstor answers it from the provider kind — thin providers only). On backends that do not guarantee it (e.g. LVM-thick) the zeros are written, so no stale data from a previously deleted volume leaks into the unwritten regions.qemu-img's--target-is-zerosupport is also probed directly (newer qemu dropped it from--helpwhile still supporting it).Types of changes
Feature/Enhancement Scale
How Has This Been Tested?
Unit tests for all new and touched wrappers/managers/orchestration:
LibvirtImportConvertedInstanceCommandWrapperTest,LibvirtConvertInstanceCommandWrapperTest,LibvirtCheckConvertInstanceCommandWrapperTest,LibvirtCheckVolumeCommandWrapperTest,the
LibvirtVmwareCbt*suites,UnmanagedVMsManagerImplTest,VolumeImportUnmanageManagerImplTest, the server-sideVmwareCbt*suites,and
VolumeOrchestratorTest— all passing.Manual/lab validation of warm CBT migration and direct-RBD import against a Ceph/RBD KVM cluster.
Linstor destination validated byte-identical on a live 3-node LINSTOR/DRBD lab against a real VMware source (cold direct-VDDK and warm CBT;
qemu-img compare= images identical), and end-to-end through a liveimportVmthat produced a Stopped CloudStack VM with its root and data disks as Linstor volumes backed by DRBD resources UpToDate on all three nodes. New Linstor unit tests across the destination gate lists, host selection, the convert/import/CBT wrappers,listPhysicalDisks, and the block-device delta/nbdcopypaths — all passing.Linstor adoption (root + data) covered by new unit tests: CheckVolume inspection via the
/dev/drbddevice path (never anrbd:URI), the in-use-on-another-node refusal, and the Linstor ⇒RAWformat resolution — all passing; live adopt-and-boot validation on the 3-node LINSTOR lab to follow.